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ABSTRACT 



This document reports on how formative evaluation was used 
by the National Institute for Science Education (NISE) to achieve its stated 
goals and to develop an institute from an amorphous group. It was found that 
formative evaluation should provide ideas, information, models, and language 
that an emerging interdisciplinary organization can use to reframe and revise 
what it is accomplishing and what kind of organization it is becoming. It 
should enhance the leaders' capacity to function as reflective practitioners 
by assisting them in tackling dialectically their most local of local detail 
and their most global of global structures. Contents include: (1) "Chronology 

of the NISE"; (2) "The Evolution and Activities of the NISE’s Formative 
Evaluation Team"; (3) "The Goals-Strategies-Outcomes Model"; (4) "Models from 
Organizational Theory"; (5) "The Knowledge Building Entity Model"; (6) "Ideal 
Intellectual Tasks"; (7) "Ideal Organizational Processes"; and (8) 
"Conclusion." (Contains 29 references.) (YDS) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



oo 

Os 



VO 

rt 



Q 



w 



JIJV87 National institute 
MWMiSjm for Science Education. 

(Mvets&y of itfsconsfrWdw&crt - Confer far Proving Scferw £& B^eotten 



Occasional Paper No. 8 

The Role of Formative Evaluation in the Development 
of an Interdisciplinary Academic Center 



Susan B. Millar 



■ i c DFPARTMENT OF EDUCATION 
Of^Fof Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 

«£ ^ CENTER (ERIC) 

m Thi«s rtnrument has been reproduced as 



originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions staled in this 
document do not necessarily .epresent 





«RIC 




Funcfed by the 

National Science Foundation 



2 



BEST COPY AVAILABLE 



National Institute for Science Education (NISE) Publications 



The NISE issues papers to facilitate the exchange of ideas among the research and 
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education and leading reformers of SMET education as found in schools, universities, 
and professional organizations across the country. The NISE Occasional Papers 
provide comment and analysis on current issues in SMET education including SMET 
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Executive Summary 



This Occasional Paper illustrates how formative evaluation helped the leaders of an 
interdisciplinary academic center — the National Institute for Science Education (NISE) — not 
only achieve a number of their original stated goals, but also develop from an amorphous group 
into an institute with many truly shared and well-defined goals, intentional organizational 
practices, and a distinctive culture. Evaluators played a critical role by providing information and 
heuristic models that helped the NISE reflect on itself and make numerous pragmatic 
improvements in organizational practices. More importantly, the process of working with the 
evaluators encouraged within the leadership a questioning and problem-solving organizational 
style that enabled them to become more reflective practitioners of the art of developing a 
productive interdisciplinary organization. 

The paper describes the formative evaluation processes used and the range of outcomes that 
resulted from these processes. Examples of the outcomes — from the specific and pragmatic to 
the conceptual and broad — are that formative evaluation helped the leaders 

• redefine their goal for including individuals from heterogeneous backgrounds-^ 

• redistribute their budget to fund fewer people for a higher percentage of their time; 

• more fully appreciate the difficulty and value of genuine interdisciplinary work; 

• improve the quality of cross-team communication processes by instituting retreats and a new 
members’ orientation packet; and 

• continuously see themselves anew, as a result of being invited to view themselves through a 
series of heuristic lenses — including “goals, strategies and outcomes,” “tightly and loosely 
coupled systems,” “forming, storming, norming, performing,” and “a knowledge building 
entity.” 

The paper suggests that other groups seeking benefits of this type should consider working with 
an evaluator. The saga that unfolds for other groups, working with other evaluators, will surely 
be different from the one told here. However, formative evaluation should, at a minimum, 
provide the following benefits to any emerging organization. 

• The experience of being interviewed by evaluators should help participants articulate their 
reasons for getting involved, the goals they hoped to achieve through their participation, and 
the relationships between these goals and the methods they were using to achieve them. 

• Formative feedback documents and other “feedback” interactions with the evaluators should 

provide the organization’s leaders with a synthesized view of the ideas and experiences 
of the various participants, and enable the leaders to perceive patterns and themes in the 
way the organization is taking shape; 

highlight situations where the goals held by the leaders and the participants are or are not 
aligned, and where implementation strategies are or are not adequate for achieving the 
stated goals, thus enabling the leaders to make effective midcourse corrections; 

invite the clients to view taken-for-granted organizational events and processes in the 
new light cast by carefully chosen or heuristic models, thereby fostering cognitive 
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conflict for the clients, and helping them notice practices and beliefs that inhibit 
achievement of their envisioned goals; and 

describe and help the leaders understand the character of the emerging organization. 

In short, formative evaluation should provide ideas, information, models, and language that an 
emerging interdisciplinary organization can use to reframe and revision what it is accomplishing 
and what kind of organization it is becoming. It should enhance the leaders’ capacity to function 
as reflective practitioners by assisting them in tacking dialectically between their most local of 
local detail and their most global of global structures. 
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Introduction 



Conflict is the gadfly of thought. It stirs us to observation and memory. It 
instigates invention. It shocks us out of sheeplike passivity, and sets us at noting 
and contriving. . . .conflict is a sine qua non of reflection and ingenuity. 

— John Dewey 



The purpose of this paper is to illustrate how formative evaluation 1 helped an interdisciplinary 
academic organization not only achieve a number of its original stated goals, but also develop 
from an amorphous group into an institute with many truly shared and well-defined goals, 
intentional organizational practices, and a distinctive culture. The paper demonstrates one way to 
encourage understanding among those who wish to improve the effectiveness of interdisciplinary 
groups. Such methods are of increasing importance because we are accelerating into a period 
when the complexity of problems in science, engineering, and social science increasingly 
demands interdisciplinary collaboration (Schrage, 1990), yet few experts are trained to create, 
and work in, effective interdisciplinary teams (Thompson-Klein, 1990). 

Formative evaluation requires both technique and art. The technique entails knowledge of and 
skill with basic social science methods of research design, data collection, and analysis 
(Atkinson, 1990; Coffey & Atkinson, 1996; Denzin & Lincoln, 1994; Miles & Huberman, 1994; 
Patton, 1990). The art, of course, lies in the wise choice of methods, the implementation of these 
methods, and the capacity to manage, and learn from, the inevitable problems arising during the 
evaluation process (Schon, 1983). 

In particular, the art of formative evaluation lies in the “continuous dialectical tacking between 
the most local of local detail and the most global of global structure in such a way as to bring 
them into simultaneous view” (Geertz, 1983, p. 69). It involves inviting the clients to peer at very 
particular and familiar organizational events and processes in the new light cast by carefully 
chosen “global structures,” “sensitizing concepts” (Blumer, 1969; Patton 1990), or heuristic 
models. That is, the evaluator asks the clients to regard themselves through the lens of a 
particular heuristic model. Typically, this “restructured viewing” fosters cognitive conflict and 
makes the familiar strange. This, in turn, leads the clients to notice — and thus begin to change — 
taken-for-granted everyday practices and beliefs that inhibit achievement of their envisioned 
goals. In other words, formative evaluation encourages the clients to develop a questioning and 
problem-solving organizational style, a habit of reflecting that quite naturally results in wise mid- 
course corrections. 

The paper is presented primarily in the first person voice of the evaluator. I take this approach in 
order to illustrate, at some level, the “tacking” process that the evaluator and, if she is successful, 
the client experience. 



1 The term formative evaluation refers to the process of providing program insiders with information about, and 
insights into, the organizational processes of their program while it is being developed in order to guide 
improvement (Joint Committee on Standards, 1994). The findings of a formative evaluation are not provided to 
parties external to the program, and are not used for accountability purposes. 
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